Channel-specific engagement machine learning architecture

ABSTRACT

A method includes generating an intervention model by determining principal components for features of a training set, associating each feature of the training set with a principal component, selecting features of the training set most highly correlated with principal components, training a machine learning model with at least some of the selected features, and saving the verified trained machine learning model as the intervention model. The method includes determining multiple channel-specific intervention expectations. Each channel-specific intervention expectation indicates a likelihood that the user will take action in response to an intervention being executed using the engagement channel corresponding to the channel-specific intervention expectation. The method includes selecting an intervention and scheduling the selected intervention for execution.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/095,504 filed Nov. 11, 2020 (now U.S. Pat. No. 11,545,260), which isa continuation-in-part of U.S. patent application Ser. No. 16/731,378filed Dec. 31, 2019 (now U.S. Pat. No. 11,551,820), which claims thebenefit of U.S. Provisional Application No. 62/787,224 filed Dec. 31,2018. The entire disclosures of the applications referenced above areincorporated by reference.

FIELD

The present disclosure relates to medical interventions and moreparticularly to systems and methods for user intervention to increaseprescription adherence.

BACKGROUND

When a user (also referred to as a patient) is prescribed medication fora condition, successful treatment of that condition requires followingthe prescription schedule. In other words, the user must fill theprescription, follow the dosage instructions, and then refill theprescription as necessary. Deviating from the dosage and refillinstructions is referred to as non-adherence.

According to some estimates, non-adherence results in $300 billion ofmedical waste every year. This medical waste may include drugs dispensedbut not taken, an increase in medical practitioner visits, and, mostparticularly, an increase in acute episodes that are much more expensiveto treat.

Because of the increased cost and worse patient outcomes caused bynon-adherence, providers in the medical space (including health insurersand pharmacy benefit managers) may perform interventions (also referredto as outreach) with users. These interventions may take the form ofphysical visits, phone calls, emails, texts, mobile alerts, etc.However, with limited resources, frequent personal outreach to everyuser may not be possible. Therefore, there is a need to develop betterintervention systems to achieve more positive patient outcomes.

The background description provided here is for the purpose of generallypresenting the context of the disclosure. Work of the presently namedinventors, to the extent it is described in this background section, aswell as aspects of the description that may not otherwise qualify asprior art at the time of filing, are neither expressly nor impliedlyadmitted as prior art against the present disclosure.

SUMMARY

A computer-implemented method includes generating an intervention modelfor a population of users based on contact data for each of thepopulation of users, demographic data for each of the population ofusers, and engagement data indicating successfulness of priorinterventions for each of the population of users. Each priorintervention corresponds to one of multiple engagement channels, and theintervention model includes multiple channel-specific models, eachcorresponding to a respective one of the multiple engagement channels.The method includes, for a first user of the population of users,obtaining first data related to the first user. The first data includesat least one of contact data of the first user, demographic data of thefirst user, and engagement data indicating successfulness of priorinterventions with the first user, and each prior intervention with thefirst user is associated with one of the multiple engagement channels.The method includes supplying the obtained first data as input to theintervention model to determine multiple channel-specific interventionexpectations. Each channel-specific intervention expectation indicates alikelihood that the first user will take action in response to anintervention being executed using the engagement channel correspondingto the channel-specific intervention expectation. The method includesdetermining a likelihood of a gap in care for the first user, and inresponse to the gap in care likelihood exceeding a minimum threshold,selecting a first intervention according to the channel-specificintervention expectation that has a highest determined value, andscheduling the selected first intervention for execution.

In other features, the multiple engagement channels include at least twoof a real-time communication with the first user by a specialist, a callto the first user by an automated call system, an email to the firstuser, and a text message to the first user. In other features, at leastone of the multiple engagement channels includes multiple interventionoptions within the engagement channel, selecting the first interventionincludes selecting a first one of the multiple intervention optionswithin the engagement channel that has a highest interventionexpectation among the intervention options, and scheduling the firstintervention includes scheduling the selected first one of the multipleintervention options within the engagement channel.

In other features, the intervention model includes a channel-agnosticintervention model that determines a general intervention expectationindicating a likelihood that the first user will take action in responseto any intervention being executed using any of the engagement channels,and in response to the general intervention expectation being below aspecified threshold, the method includes initiating a low engagementintervention process and identifying at least one reason for lowengagement of the first user.

In other features, the method includes, in response to determining thata time elapsed since a most recent intervention for the first user isless than a specified delay threshold, waiting until the specified delaythreshold has elapsed prior to selecting the first intervention. Inother features, the method includes identifying one or more targetsrelevant to the first user, determining a measure of progress toward atleast one of the identified targets, determining an engagementimportance metric based on the determined measure of progress, andweighting the channel-specific intervention expectations according tothe determined engagement importance metric prior to selecting the firstintervention.

In other features, scheduling the first intervention includes, inresponse to determining that the measure of progress is less than aspecified minimum threshold, scheduling an intervention corresponding tothe channel-specific intervention expectation that has a highestdetermined value prior to weighting. In other features, the methodincludes determining a cost of engagement for each of the multipleengagement channels, determining a channel capacity for each of themultiple engagement channels, and weighting the channel-specificintervention expectations according to the determined costs ofengagement and channel capacities prior to selecting the firstintervention.

In other features, the method includes determining at least one useradherence factor associated with the first user, and weighting thechannel-specific intervention expectations according to the determineduser adherence factor prior to selecting the first intervention. Inother features, the at least one user adherence factor includes at leastone of a prescription refill cost to the first user, a pill burden ofthe first user, and a comorbidity condition of the first user.

In other features, scheduling the first intervention includes, inresponse to the comorbidity condition of the first user indicating afuture adverse event risk that is higher than a specified threshold,scheduling an intervention corresponding to the channel-specificintervention expectation that has a highest determined value prior toweighting. In other features, the method includes tracking success ofthe selected first intervention based on a medication possession ratio(MPR). The MPR is calculated as a number of days of a prescriptioncurrently in possession of the first user divided by a number of totaldays initially supplied with the prescription.

A computer system includes memory configured to storecomputer-executable instructions and an intervention model for apopulation of users based on contact data for each of the population ofusers, demographic data for each of the population of users, andengagement data indicating successfulness of prior interventions foreach of the population of users. Each prior intervention corresponds toone of multiple engagement channels, and the intervention model includesmultiple channel-specific models, each corresponding to a respective oneof the multiple engagement channels. The system includes at least oneprocessor configured to execute the instructions. The instructionsinclude, for a first user of the population of users, obtaining firstdata related to the first user. The first data includes at least one ofcontact data of the first user, demographic data of the first user, andengagement data indicating successfulness of prior interventions withthe first user, and each prior intervention with the first user isassociated with one of the multiple engagement channels. Theinstructions include supplying the obtained first data as input to theintervention model to determine multiple channel-specific interventionexpectations. Each channel-specific intervention expectation indicates alikelihood that the first user will take action in response to anintervention being executed using the engagement channel correspondingto the channel-specific intervention expectation. The instructionsinclude determining a likelihood of a gap in care for the first user,and in response to the gap in care likelihood exceeding a minimumthreshold, selecting a first intervention according to thechannel-specific intervention expectation that has a highest determinedvalue, and scheduling the selected first intervention for execution.

In other features, the multiple engagement channels include at least twoof a real-time communication with the first user by a specialist, a callto the first user by an automated call system, an email to the firstuser, and a text message to the first user. In other features, at leastone of the multiple engagement channels includes multiple interventionoptions within the engagement channel, selecting the first interventionincludes selecting a first one of the multiple intervention optionswithin the engagement channel that has a highest interventionexpectation among the intervention options, and scheduling the firstintervention includes scheduling the selected first one of the multipleintervention options within the engagement channel.

In other features, the intervention model includes a channel-agnosticintervention model that determines a general intervention expectationindicating a likelihood that the first user will take action in responseto any intervention being executed using any of the engagement channels,and in response to the general intervention expectation being below aspecified threshold, the instructions include initiating a lowengagement intervention process and identifying at least one reason forlow engagement of the first user.

In other features, the instructions further include, in response todetermining that a time elapsed since a most recent intervention for thefirst user is less than a specified delay threshold, waiting until thespecified delay threshold has elapsed prior to selecting the firstintervention. In other features, the instructions further includeidentifying one or more targets relevant to the first user, determininga measure of progress toward at least one of the identified targets,determining an engagement importance metric based on the determinedmeasure of progress, and weighting the channel-specific interventionexpectations according to the determined engagement importance metricprior to selecting the first intervention.

In other features, scheduling the first intervention includes, inresponse to determining that the measure of progress is less than aspecified minimum threshold, scheduling an intervention corresponding tothe channel-specific intervention expectation that has a highestdetermined value prior to weighting.

In other features, the instructions further include determining a costof engagement for each of the multiple engagement channels, determininga channel capacity for each of the multiple engagement channels, andweighting the channel-specific intervention expectations according tothe determined costs and channel capacities prior to selecting the firstintervention.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description, the claims, and the drawings.The detailed description and specific examples are intended for purposesof illustration only and are not intended to limit the scope of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings.

FIG. 1 is a functional block diagram of an example system including ahigh-volume pharmacy.

FIG. 2 is a functional block diagram of an example pharmacy fulfillmentdevice, which may be deployed within the system of FIG. 1 .

FIG. 3 is a functional block diagram of an example order processingdevice, which may be deployed within the system of FIG. 1 .

FIG. 4 is a functional block diagram of an example implementation of anintervention device according to the principles of the presentdisclosure.

FIG. 5 is a functional block diagram of an example intervention modelingcircuit.

FIG. 6 is a functional block diagram of another example implementationof an intervention modeling circuit.

FIG. 7 is a flowchart showing example intervention model creation andmaintenance.

FIG. 8 is a flowchart showing example intervention determination for aspecified user.

FIGS. 9A, 9B, 9C, 9D, and 9E are graphical representations of exampleintervention scripts for real-time interventions.

FIG. 10 is a functional block diagram of an example implementation of achannel-specific engagement machine learning architecture according tothe principles of the present disclosure.

FIG. 11 is a functional block diagram of example channel-specificintervention modeling circuits.

FIG. 12 is a functional block diagram of an example interventionmanagement circuit.

FIGS. 13A and 13B together are a flowchart showing examplechannel-specific intervention determination for a specified user.

In the drawings, reference numbers may be reused to identify similarand/or identical elements.

DETAILED DESCRIPTION Introduction

The present disclosure describes how an operator in the medical spacecan use an intervention system to identify users for whom interventionsare necessary and to select the appropriate interventions for specificusers. Interventions may be warranted when it appears likely that therewill be a gap in care, such as a medication running out prior toconclusion of a treatment regimen. In one specific example, a gap incare is experienced when a user exhausts their supply of prescriptionpills prior to obtaining a refill of the prescription.

Interventions may take the form of an operator in the medical space(such as a health insurer or a pharmacy benefit manager) contacting theuser. For example, an operator may directly contact the user or requestthat a local pharmacy, medical provider, or caregiver contact the user.These contacts may take the form of personal visits, telephone calls,text messages, mobile alerts, emails, postal letters, etc.Communications with the user may include reminders about their course oftreatment, including expected dates by which existing medication will beexhausted, expired, or otherwise need refilling.

The communications may also include warnings about the potential effectsof a gap in care. The communications may provide incentives for the userto avoid a gap in care, such as discounts on drugs, free shipping, ordiscounts on expedited shipping. In addition, the communications mayassist the user in setting up automatic refills and other technologicalapproaches to increasing adherence. Another technological approach toincrease adherence is establishing mail-order prescriptions, which mayreduce the time and transportation barriers to obtaining new and refillprescriptions from a retail pharmacy.

In addition to choosing between these types of interventions, specificsabout the interventions may be determined. For example, a time of day atwhich to make the intervention may be specified. In addition, whenmultiple contact methods are available, the type of contact (such aswork email, home phone, etc.) may be selected along with the time of dayand day of week for the intervention.

A system according to the present disclosure identifies which users areat risk of a gap in care and also how receptive the users will be tointerventions. In this way, interventions can be targeted to havemaximum impact. In some implementations, the cost or patient outcomeassociated with non-adherence may be taken into account so thatinterventions can be directed to those users where non-adherence has amore drastic negative outcome for the patient and/or may incur a highercost in treatment due to a gap in care.

As one example only, a two-by-two matrix may be used to selectinterventions, with users at high risk of a gap in care and a highlikelihood of engagement with an intervention will receive a real-timeindication (such as a phone call). Meanwhile, users at high risk of agap in care but a low likelihood of engagement with an intervention mayreceive a physical letter with a discount code for mail-ordering aprescription. Users with a lower risk of a gap in care may receive atext message, regardless of whether their likelihood of engagement ishigh or low. In other words, in this particular example of potentialbusiness logic, two of the cells in this matrix are assigned the sameintervention. Users whose risk of a gap in care is low enough receive nointervention at all and are therefore not reflected in this matrix.

In other implementations, a single score may be determined based on twoor more of (i) risk of gap in care, (ii) likelihood of engagement, and(iii) cost of gap in care. Then, ranges of this single score areassigned various interventions. Values of the single score below acertain threshold (for example, indicating both a low risk of a gap incare as well as a low likelihood of engagement with interventions) maybe assigned no intervention.

A variety of models may be implemented to estimate risks of a gap incare. In various implementations, multiple models may be implemented andone or more models may be selected based on user data. For example, afirst model may estimate the risk of a user experiencing a gap in carerelated to treatment for diabetes. A second model may estimate a risk ofa gap in care for treatment related to hypertension. A third model mayestimate a risk of a gap in care for treatment of high cholesterol. Afourth model may estimate a risk of a high figure of merit beingmeasured (such as A1C for a diabetic). A fifth model may estimate a riskof a user not obtaining a lab test for a particular condition (such as ablood test for ongoing diabetes management).

Meanwhile, a general intervention model may be implemented to determinethe likelihood of a user engaging with an executed intervention. Theintervention model may be agnostic to the condition of the patient,meaning that the intervention model is applicable to patients havingdiabetes, hypertension, high cholesterol, multiple of or none of thoseconditions. In addition, the intervention model may be agnostic to thetype (or, channel) of engagement. In other words, the intervention modelprovides an estimate of the user engaging with an interventionregardless of the form of the intervention (such as a phone call or anemail).

In some implementations, additional specific models may be implemented.For example, models may be implemented that are specific to the medicalcondition of the user. For example, a diabetes model may be implementedfor diabetic users. Meanwhile, a hypertension model may be implementedfor hypertensive users and a lipid model may be implemented for userswith high cholesterol. A priority ranking system may select which modelis applied to the user that has multiple conditions. In otherimplementations, all applicable models may be applied to the user and asingle score calculated based on a weighted combination or a votingscheme.

In various implementations, channel-specific intervention models may bedeveloped. These channel-specific intervention models may assess thelikelihood that the user will engage with, for example, a phone callversus an email. Some of these models may be restricted to specificusers or to specific populations or groups of users. When multiple ofthe models apply to a specific user, different engagement scores may bedetermined for a single user and are then used to determine whichintervention to execute for that user.

Based on the risk models and intervention models, an operator candetermine which users require intervention and select the appropriateinterventions for the users. The operator can then queue theseinterventions, such as by scheduling emails for transmission and addingphone calls to a schedule for specialists (such as pharmacists). Asthese interventions are performed (such as emails being sent or callsbeing made), the fact of the intervention as well as the feedback fromthe user regarding intervention is logged.

In addition, over time, the success of the intervention may be trackedbased on whether the user fills required prescriptions within a timelymanner. One objective measure of success may be medication possessionratio (MPR), which may be calculated by the number of days of aprescription currently in the user's possession divided by the number oftotal days initially supplied with the prescription. A higher MPRindicates that the user is less likely to exhaust their supply of thedrug.

A client of the operator, such as a health plan or medical practitioner,may access data about executed interventions (transmitted emails, placedphone calls, etc.) for their users. A web portal may allow the client(such as a health insurance representative) to review pastinterventions, execute their own interventions (such as placing a callto the user's phone number), and log the interventions performed by theclient. In addition, the client may be able to access data about thelikelihood of engagement by the user so that the client can make theirown decisions, based on business logic, regarding executinginterventions for particular users.

High-Volume Pharmacy

FIG. 1 is a block diagram of an example implementation of a system 100for a high-volume pharmacy. While the system 100 is generally describedas being deployed in a high-volume pharmacy or a fulfillment center (forexample, a mail order pharmacy, a direct delivery pharmacy, etc.), thesystem 100 and/or components of the system 100 may otherwise be deployed(for example, in a lower-volume pharmacy, etc.). A high-volume pharmacymay be a pharmacy that is capable of filling at least some prescriptionsmechanically. The system 100 may include a benefit manager device 102and a pharmacy device 106 in communication with each other directlyand/or over a network 104.

The system 100 may also include one or more user device(s) 108. A user,such as a pharmacist, patient, data analyst, health plan administrator,etc., may access the benefit manager device 102 or the pharmacy device106 using the user device 108. The user device 108 may be a desktopcomputer, a laptop computer, a tablet, a smartphone, etc.

The benefit manager device 102 is a device operated by an entity that isat least partially responsible for creation and/or management of thepharmacy or drug benefit. While the entity operating the benefit managerdevice 102 is typically a pharmacy benefit manager (PBM), other entitiesmay operate the benefit manager device 102 on behalf of themselves orother entities (such as PBMs). For example, the benefit manager device102 may be operated by a health plan, a retail pharmacy chain, a drugwholesaler, a data analytics or other type of software-related company,etc. In some implementations, a PBM that provides the pharmacy benefitmay provide one or more additional benefits including a medical orhealth benefit, a dental benefit, a vision benefit, a wellness benefit,a radiology benefit, a pet care benefit, an insurance benefit, a longterm care benefit, a nursing home benefit, etc. The PBM may, in additionto its PBM operations, operate one or more pharmacies. The pharmaciesmay be retail pharmacies, mail order pharmacies, etc.

Some of the operations of the PBM that operates the benefit managerdevice 102 may include the following activities and processes. A member(or a person on behalf of the member) of a pharmacy benefit plan mayobtain a prescription drug at a retail pharmacy location (e.g., alocation of a physical store) from a pharmacist or a pharmacisttechnician. The member may also obtain the prescription drug throughmail order drug delivery from a mail order pharmacy location, such asthe system 100. In some implementations, the member may obtain theprescription drug directly or indirectly through the use of a machine,such as a kiosk, a vending unit, a mobile electronic device, or adifferent type of mechanical device, electrical device, electroniccommunication device, and/or computing device. Such a machine may befilled with the prescription drug in prescription packaging, which mayinclude multiple prescription components, by the system 100. Thepharmacy benefit plan is administered by or through the benefit managerdevice 102.

The member may have a copayment for the prescription drug that reflectsan amount of money that the member is responsible to pay the pharmacyfor the prescription drug. The money paid by the member to the pharmacymay come from, as examples, personal funds of the member, a healthsavings account (HSA) of the member or the member's family, a healthreimbursement arrangement (HRA) of the member or the member's family, ora flexible spending account (FSA) of the member or the member's family.In some instances, an employer of the member may directly or indirectlyfund or reimburse the member for the copayments.

The amount of the copayment required by the member may vary acrossdifferent pharmacy benefit plans having different plan sponsors orclients and/or for different prescription drugs. The member's copaymentmay be a flat copayment (in one example, $10), coinsurance (in oneexample, 10%), and/or a deductible (for example, responsibility for thefirst $500 of annual prescription drug expense, etc.) for certainprescription drugs, certain types and/or classes of prescription drugs,and/or all prescription drugs. The copayment may be stored in a storagedevice 110 or determined by the benefit manager device 102.

In some instances, the member may not pay the copayment or may only paya portion of the copayment for the prescription drug. For example, if ausual and customary cost for a generic version of a prescription drug is$4, and the member's flat copayment is $20 for the prescription drug,the member may only need to pay $4 to receive the prescription drug. Inanother example involving a worker's compensation claim, no copaymentmay be due by the member for the prescription drug.

In addition, copayments may also vary based on different deliverychannels for the prescription drug. For example, the copayment forreceiving the prescription drug from a mail order pharmacy location maybe less than the copayment for receiving the prescription drug from aretail pharmacy location.

In conjunction with receiving a copayment (if any) from the member anddispensing the prescription drug to the member, the pharmacy submits aclaim to the PBM for the prescription drug. After receiving the claim,the PBM (such as by using the benefit manager device 102) may performcertain adjudication operations including verifying eligibility for themember, identifying/reviewing an applicable formulary for the member todetermine any appropriate copayment, coinsurance, and deductible for theprescription drug, and performing a drug utilization review (DUR) forthe member. Further, the PBM may provide a response to the pharmacy (forexample, the pharmacy system 100) following performance of at least someof the aforementioned operations.

As part of the adjudication, a plan sponsor (or the PBM on behalf of theplan sponsor) ultimately reimburses the pharmacy for filling theprescription drug when the prescription drug was successfullyadjudicated. The aforementioned adjudication operations generally occurbefore the copayment is received and the prescription drug is dispensed.However in some instances, these operations may occur simultaneously,substantially simultaneously, or in a different order. In addition, moreor fewer adjudication operations may be performed as at least part ofthe adjudication process.

The amount of reimbursement paid to the pharmacy by a plan sponsorand/or money paid by the member may be determined at least partiallybased on types of pharmacy networks in which the pharmacy is included.In some implementations, the amount may also be determined based onother factors. For example, if the member pays the pharmacy for theprescription drug without using the prescription or drug benefitprovided by the PBM, the amount of money paid by the member may behigher than when the member uses the prescription or drug benefit. Insome implementations, the amount of money received by the pharmacy fordispensing the prescription drug and for the prescription drug itselfmay be higher than when the member uses the prescription or drugbenefit. Some or all of the foregoing operations may be performed byexecuting instructions stored in the benefit manager device 102 and/oran additional device.

Examples of the network 104 include a Global System for MobileCommunications (GSM) network, a code division multiple access (CDMA)network, 3rd Generation Partnership Project (3GPP), an Internet Protocol(IP) network, a Wireless Application Protocol (WAP) network, or an IEEE802.11 standards network, as well as various combinations of the abovenetworks. The network 104 may include an optical network. The network104 may be a local area network or a global communication network, suchas the Internet. In some implementations, the network 104 may include anetwork dedicated to prescription orders: a prescribing network such asthe electronic prescribing network operated by Surescripts of Arlington,Virginia.

Moreover, although the system shows a single network 104, multiplenetworks can be used. The multiple networks may communicate in seriesand/or parallel with each other to link the devices 102-110.

The pharmacy device 106 may be a device associated with a retailpharmacy location (e.g., an exclusive pharmacy location, a grocery storewith a retail pharmacy, or a general sales store with a retail pharmacy)or other type of pharmacy location at which a member attempts to obtaina prescription. The pharmacy may use the pharmacy device 106 to submitthe claim to the PBM for adjudication.

Additionally, in some implementations, the pharmacy device 106 mayenable information exchange between the pharmacy and the PBM. Forexample, this may allow the sharing of member information such as drughistory that may allow the pharmacy to better service a member (forexample, by providing more informed therapy consultation and druginteraction information). In some implementations, the benefit managerdevice 102 may track prescription drug fulfillment and/or otherinformation for users that are not members, or have not identifiedthemselves as members, at the time (or in conjunction with the time) inwhich they seek to have a prescription filled at a pharmacy.

The pharmacy device 106 may include a pharmacy fulfillment device 112,an order processing device 114, and a pharmacy management device 116 incommunication with each other directly and/or over the network 104. Theorder processing device 114 may receive information regarding fillingprescriptions and may direct an order component to one or more devicesof the pharmacy fulfillment device 112 at a pharmacy. The pharmacyfulfillment device 112 may fulfill, dispense, aggregate, and/or pack theorder components of the prescription drugs in accordance with one ormore prescription orders directed by the order processing device 114.

In general, the order processing device 114 is a device located withinor otherwise associated with the pharmacy to enable the pharmacyfulfillment device 112 to fulfill a prescription and dispenseprescription drugs. In some implementations, the order processing device114 may be an external order processing device separate from thepharmacy and in communication with other devices located within thepharmacy.

For example, the external order processing device may communicate withan internal pharmacy order processing device and/or other deviceslocated within the system 100. In some implementations, the externalorder processing device may have limited functionality (e.g., asoperated by a user requesting fulfillment of a prescription drug), whilethe internal pharmacy order processing device may have greaterfunctionality (e.g., as operated by a pharmacist).

The order processing device 114 may track the prescription order as itis fulfilled by the pharmacy fulfillment device 112. The prescriptionorder may include one or more prescription drugs to be filled by thepharmacy. The order processing device 114 may make pharmacy routingdecisions and/or order consolidation decisions for the particularprescription order. The pharmacy routing decisions include whatdevice(s) in the pharmacy are responsible for filling or otherwisehandling certain portions of the prescription order. The orderconsolidation decisions include whether portions of one prescriptionorder or multiple prescription orders should be shipped together for auser or a user family. The order processing device 114 may also trackand/or schedule literature or paperwork associated with eachprescription order or multiple prescription orders that are beingshipped together. In some implementations, the order processing device114 may operate in combination with the pharmacy management device 116.

The order processing device 114 may include circuitry, a processor, amemory to store data and instructions, and communication functionality.The order processing device 114 is dedicated to performing processes,methods, and/or instructions described in this application. Other typesof electronic devices may also be used that are specifically configuredto implement the processes, methods, and/or instructions described infurther detail below.

In some implementations, at least some functionality of the orderprocessing device 114 may be included in the pharmacy management device116. The order processing device 114 may be in a client-serverrelationship with the pharmacy management device 116, in a peer-to-peerrelationship with the pharmacy management device 116, or in a differenttype of relationship with the pharmacy management device 116. The orderprocessing device 114 and/or the pharmacy management device 116 maycommunicate directly (for example, such as by using a local storage)and/or through the network 104 (such as by using a cloud storageconfiguration, software as a service, etc.) with the storage device 110.

The storage device 110 may include: non-transitory storage (for example,memory, hard disk, CD-ROM, etc.) in communication with the benefitmanager device 102 and/or the pharmacy device 106 directly and/or overthe network 104. The non-transitory storage may store order data 118,member data 120, claims data 122, drug data 124, prescription data 126,and/or plan sponsor data 128. Further, the system 100 may includeadditional devices, which may communicate with each other directly orover the network 104.

The order data 118 may be related to a prescription order. The orderdata may include type of the prescription drug (for example, drug nameand strength) and quantity of the prescription drug. The order data 118may also include data used for completion of the prescription, such asprescription materials. In general, prescription materials include anelectronic copy of information regarding the prescription drug forinclusion with or otherwise in conjunction with the fulfilledprescription. The prescription materials may include electronicinformation regarding drug interaction warnings, recommended usage,possible side effects, expiration date, date of prescribing, etc. Theorder data 118 may be used by a high-volume fulfillment center tofulfill a pharmacy order.

In some implementations, the order data 118 includes verificationinformation associated with fulfillment of the prescription in thepharmacy. For example, the order data 118 may include videos and/orimages taken of (i) the prescription drug prior to dispensing, duringdispensing, and/or after dispensing, (ii) the prescription container(for example, a prescription container and sealing lid, prescriptionpackaging, etc.) used to contain the prescription drug prior todispensing, during dispensing, and/or after dispensing, (iii) thepackaging and/or packaging materials used to ship or otherwise deliverthe prescription drug prior to dispensing, during dispensing, and/orafter dispensing, and/or (iv) the fulfillment process within thepharmacy. Other types of verification information such as barcode dataread from pallets, bins, trays, or carts used to transport prescriptionswithin the pharmacy may also be stored as order data 118.

The member data 120 includes information regarding the membersassociated with the PBM. The information stored as member data 120 mayinclude personal information, personal health information, protectedhealth information, etc. Examples of the member data 120 include name,address, telephone number, e-mail address, prescription drug history,etc. The member data 120 may include a plan sponsor identifier thatidentifies the plan sponsor associated with the member and/or a memberidentifier that identifies the member to the plan sponsor. The memberdata 120 may include a member identifier that identifies the plansponsor associated with the user and/or a user identifier thatidentifies the user to the plan sponsor. The member data 120 may alsoinclude dispensation preferences such as type of label, type of cap,message preferences, language preferences, etc.

The member data 120 may be accessed by various devices in the pharmacy(for example, the high-volume fulfillment center, etc.) to obtaininformation used for fulfillment and shipping of prescription orders. Insome implementations, an external order processing device operated by oron behalf of a member may have access to at least a portion of themember data 120 for review, verification, or other purposes.

In some implementations, the member data 120 may include information forpersons who are users of the pharmacy but are not members in thepharmacy benefit plan being provided by the PBM. For example, theseusers may obtain drugs directly from the pharmacy, through a privatelabel service offered by the pharmacy, the high-volume fulfillmentcenter, or otherwise. In general, the use of the terms “member” and“user” may be used interchangeably.

The claims data 122 includes information regarding pharmacy claimsadjudicated by the PBM under a drug benefit program provided by the PBMfor one or more plan sponsors. In general, the claims data 122 includesan identification of the client that sponsors the drug benefit programunder which the claim is made, and/or the member that purchased theprescription drug giving rise to the claim, the prescription drug thatwas filled by the pharmacy (e.g., the national drug code number, etc.),the dispensing date, generic indicator, generic product identifier (GPI)number, medication class, the cost of the prescription drug providedunder the drug benefit program, the copayment/coinsurance amount, rebateinformation, and/or member eligibility, etc. Additional information maybe included.

In some implementations, other types of claims beyond prescription drugclaims may be stored in the claims data 122. For example, medicalclaims, dental claims, wellness claims, or other types ofhealth-care-related claims for members may be stored as a portion of theclaims data 122.

In some implementations, the claims data 122 includes claims thatidentify the members with whom the claims are associated. Additionallyor alternatively, the claims data 122 may include claims that have beende-identified (that is, associated with a unique identifier but not witha particular, identifiable member).

The drug data 124 may include drug name (e.g., technical name and/orcommon name), other names by which the drug is known, activeingredients, an image of the drug (such as in pill form), etc. The drugdata 124 may include information associated with a single medication ormultiple medications.

The prescription data 126 may include information regardingprescriptions that may be issued by prescribers on behalf of users, whomay be members of the pharmacy benefit plan—for example, to be filled bya pharmacy. Examples of the prescription data 126 include user names,medication or treatment (such as lab tests), dosing information, etc.The prescriptions may include electronic prescriptions or paperprescriptions that have been scanned. In some implementations, thedosing information reflects a frequency of use (e.g., once a day, twicea day, before each meal, etc.) and a duration of use (e.g., a few days,a week, a few weeks, a month, etc.).

In some implementations, the order data 118 may be linked to associatedmember data 120, claims data 122, drug data 124, and/or prescriptiondata 126.

The plan sponsor data 128 includes information regarding the plansponsors of the PBM. Examples of the plan sponsor data 128 includecompany name, company address, contact name, contact telephone number,contact e-mail address, etc.

FIG. 2 illustrates the pharmacy fulfillment device 112 according to anexample implementation. The pharmacy fulfillment device 112 may be usedto process and fulfill prescriptions and prescription orders. Afterfulfillment, the fulfilled prescriptions are packed for shipping.

The pharmacy fulfillment device 112 may include devices in communicationwith the benefit manager device 102, the order processing device 114,and/or the storage device 110, directly or over the network 104.Specifically, the pharmacy fulfillment device 112 may include palletsizing and pucking device(s) 206, loading device(s) 208, inspectdevice(s) 210, unit of use device(s) 212, automated dispensing device(s)214, manual fulfillment device(s) 216, review devices 218, imagingdevice(s) 220, cap device(s) 222, accumulation devices 224, packingdevice(s) 226, literature device(s) 228, unit of use packing device(s)230, and mail manifest device(s) 232. Further, the pharmacy fulfillmentdevice 112 may include additional devices, which may communicate witheach other directly or over the network 104.

In some implementations, operations performed by one of these devices206-232 may be performed sequentially, or in parallel with theoperations of another device as may be coordinated by the orderprocessing device 114. In some implementations, the order processingdevice 114 tracks a prescription with the pharmacy based on operationsperformed by one or more of the devices 206-232.

In some implementations, the pharmacy fulfillment device 112 maytransport prescription drug containers, for example, among the devices206-232 in the high-volume fulfillment center, by use of pallets. Thepallet sizing and pucking device 206 may configure pucks in a pallet. Apallet may be a transport structure for a number of prescriptioncontainers, and may include a number of cavities. A puck may be placedin one or more than one of the cavities in a pallet by the pallet sizingand pucking device 206. The puck may include a receptacle sized andshaped to receive a prescription container. Such containers may besupported by the pucks during carriage in the pallet. Different pucksmay have differently sized and shaped receptacles to accommodatecontainers of differing sizes, as may be appropriate for differentprescriptions.

The arrangement of pucks in a pallet may be determined by the orderprocessing device 114 based on prescriptions that the order processingdevice 114 decides to launch. The arrangement logic may be implementeddirectly in the pallet sizing and pucking device 206. Once aprescription is set to be launched, a puck suitable for the appropriatesize of container for that prescription may be positioned in a pallet bya robotic arm or pickers. The pallet sizing and pucking device 206 maylaunch a pallet once pucks have been configured in the pallet.

The loading device 208 may load prescription containers into the puckson a pallet by a robotic arm, a pick and place mechanism (also referredto as pickers), etc. In various implementations, the loading device 208has robotic arms or pickers to grasp a prescription container and moveit to and from a pallet or a puck. The loading device 208 may also printa label that is appropriate for a container that is to be loaded ontothe pallet, and apply the label to the container. The pallet may belocated on a conveyor assembly during these operations (e.g., at thehigh-volume fulfillment center, etc.).

The inspect device 210 may verify that containers in a pallet arecorrectly labeled and in the correct spot on the pallet. The inspectdevice 210 may scan the label on one or more containers on the pallet.Labels of containers may be scanned or imaged in full or in part by theinspect device 210. Such imaging may occur after the container has beenlifted out of its puck by a robotic arm, picker, etc., or may beotherwise scanned or imaged while retained in the puck. In someimplementations, images and/or video captured by the inspect device 210may be stored in the storage device 110 as order data 118.

The unit of use device 212 may temporarily store, monitor, label, and/ordispense unit of use products. In general, unit of use products areprescription drug products that may be delivered to a user or memberwithout being repackaged at the pharmacy. These products may includepills in a container, pills in a blister pack, inhalers, etc.Prescription drug products dispensed by the unit of use device 212 maybe packaged individually or collectively for shipping, or may be shippedin combination with other prescription drugs dispensed by other devicesin the high-volume fulfillment center.

At least some of the operations of the devices 206-232 may be directedby the order processing device 114. For example, the manual fulfillmentdevice 216, the review device 218, the automated dispensing device 214,and/or the packing device 226, etc. may receive instructions provided bythe order processing device 114.

The automated dispensing device 214 may include one or more devices thatdispense prescription drugs or pharmaceuticals into prescriptioncontainers in accordance with one or multiple prescription orders. Ingeneral, the automated dispensing device 214 may include mechanical andelectronic components with, in some implementations, software and/orlogic to facilitate pharmaceutical dispensing that would otherwise beperformed in a manual fashion by a pharmacist and/or pharmacisttechnician. For example, the automated dispensing device 214 may includehigh-volume fillers that fill a number of prescription drug types at arapid rate and blister pack machines that dispense and pack drugs into ablister pack. Prescription drugs dispensed by the automated dispensingdevices 214 may be packaged individually or collectively for shipping,or may be shipped in combination with other prescription drugs dispensedby other devices in the high-volume fulfillment center.

The manual fulfillment device 216 controls how prescriptions aremanually fulfilled. For example, the manual fulfillment device 216 mayreceive or obtain a container and enable fulfillment of the container bya pharmacist or pharmacy technician. In some implementations, the manualfulfillment device 216 provides the filled container to another devicein the pharmacy fulfillment devices 112 to be joined with othercontainers in a prescription order for a user or member.

In general, manual fulfillment may include operations at least partiallyperformed by a pharmacist or a pharmacy technician. For example, aperson may retrieve a supply of the prescribed drug, may make anobservation, may count out a prescribed quantity of drugs and place theminto a prescription container, etc. Some portions of the manualfulfillment process may be automated by use of a machine. For example,counting of capsules, tablets, or pills may be at least partiallyautomated (such as through use of a pill counter). Prescription drugsdispensed by the manual fulfillment device 216 may be packagedindividually or collectively for shipping, or may be shipped incombination with other prescription drugs dispensed by other devices inthe high-volume fulfillment center.

The review device 218 may process prescription containers to be reviewedby a pharmacist for proper pill count, exception handling, prescriptionverification, etc. Fulfilled prescriptions may be manually reviewedand/or verified by a pharmacist, as may be required by state or locallaw. A pharmacist or other licensed pharmacy person who may dispensecertain drugs in compliance with local and/or other laws may operate thereview device 218 and visually inspect a prescription container that hasbeen filled with a prescription drug. The pharmacist may review, verify,and/or evaluate drug quantity, drug strength, and/or drug interactionconcerns, or otherwise perform pharmacist services. The pharmacist mayalso handle containers which have been flagged as an exception, such ascontainers with unreadable labels, containers for which the associatedprescription order has been canceled, containers with defects, etc. Inan example, the manual review can be performed at a manual reviewstation.

The imaging device 220 may image containers once they have been filledwith pharmaceuticals. The imaging device 220 may measure a fill heightof the pharmaceuticals in the container based on the obtained image todetermine if the container is filled to the correct height given thetype of pharmaceutical and the number of pills in the prescription.Images of the pills in the container may also be obtained to detect thesize of the pills themselves and markings thereon. The images may betransmitted to the order processing device 114 and/or stored in thestorage device 110 as part of the order data 118.

The cap device 222 may be used to cap or otherwise seal a prescriptioncontainer. In some implementations, the cap device 222 may secure aprescription container with a type of cap in accordance with a userpreference (e.g., a preference regarding child resistance, etc.), a plansponsor preference, a prescriber preference, etc. The cap device 222 mayalso etch a message into the cap, although this process may be performedby a subsequent device in the high-volume fulfillment center.

The accumulation device 224 accumulates various containers ofprescription drugs in a prescription order. The accumulation device 224may accumulate prescription containers from various devices or areas ofthe pharmacy. For example, the accumulation device 224 may accumulateprescription containers from the unit of use device 212, the automateddispensing device 214, the manual fulfillment device 216, and the reviewdevice 218. The accumulation device 224 may be used to group theprescription containers prior to shipment to the member.

The literature device 228 prints, or otherwise generates, literature toinclude with each prescription drug order. The literature may be printedon multiple sheets of substrates, such as paper, coated paper, printablepolymers, or combinations of the above substrates. The literatureprinted by the literature device 228 may include information required toaccompany the prescription drugs included in a prescription order, otherinformation related to prescription drugs in the order, financialinformation associated with the order (for example, an invoice or anaccount statement), etc.

In some implementations, the literature device 228 folds or otherwiseprepares the literature for inclusion with a prescription drug order(e.g., in a shipping container). In other implementations, theliterature device 228 prints the literature and is separate from anotherdevice that prepares the printed literature for inclusion with aprescription order.

The packing device 226 packages the prescription order in preparationfor shipping the order. The packing device 226 may box, bag, orotherwise package the fulfilled prescription order for delivery. Thepacking device 226 may further place inserts (e.g., literature or otherpapers, etc.) into the packaging received from the literature device228. For example, bulk prescription orders may be shipped in a box,while other prescription orders may be shipped in a bag, which may be awrap seal bag.

The packing device 226 may label the box or bag with an address and arecipient's name. The label may be printed and affixed to the bag orbox, be printed directly onto the bag or box, or otherwise associatedwith the bag or box. The packing device 226 may sort the box or bag formailing in an efficient manner (e.g., sort by delivery address, etc.).The packing device 226 may include ice or temperature sensitive elementsfor prescriptions that are to be kept within a temperature range duringshipping (for example, this may be necessary in order to retainefficacy). The ultimate package may then be shipped through postal mail,through a mail order delivery service that ships via ground and/or air(e.g., UPS, FEDEX, or DHL, etc.), through a delivery service, through alocker box at a shipping site (e.g., AMAZON locker or a PO Box, etc.),or otherwise.

The unit of use packing device 230 packages a unit of use prescriptionorder in preparation for shipping the order. The unit of use packingdevice 230 may include manual scanning of containers to be bagged forshipping to verify each container in the order. In an exampleimplementation, the manual scanning may be performed at a manualscanning station. The pharmacy fulfillment device 112 may also include amail manifest device 232 to print mailing labels used by the packingdevice 226 and may print shipping manifests and packing lists.

While the pharmacy fulfillment device 112 in FIG. 2 is shown to includesingle devices 206-232, multiple devices may be used. When multipledevices are present, the multiple devices may be of the same device typeor models, or may be a different device type or model. The types ofdevices 206-232 shown in FIG. 2 are example devices. In otherconfigurations of the system 100, lesser, additional, or different typesof devices may be included.

Moreover, multiple devices may share processing and/or memory resources.The devices 206-232 may be located in the same area or in differentlocations. For example, the devices 206-232 may be located in a buildingor set of adjoining buildings. The devices 206-232 may be interconnected(such as by conveyors), networked, and/or otherwise in contact with oneanother or integrated with one another (e.g., at the high-volumefulfillment center, etc.). In addition, the functionality of a devicemay be split among a number of discrete devices and/or combined withother devices.

FIG. 3 illustrates the order processing device 114 according to anexample implementation. The order processing device 114 may be used byone or more operators to generate prescription orders, make routingdecisions, make prescription order consolidation decisions, trackliterature with the system 100, and/or view order status and other orderrelated information. For example, the prescription order may becomprised of order components.

The order processing device 114 may receive instructions to fulfill anorder without operator intervention. An order component may include aprescription drug fulfilled by use of a container through the system100. The order processing device 114 may include an order verificationsubsystem 302, an order control subsystem 304, and/or an order trackingsubsystem 306. Other subsystems may also be included in the orderprocessing device 114.

The order verification subsystem 302 may communicate with the benefitmanager device 102 to verify the eligibility of the member and reviewthe formulary to determine appropriate copayment, coinsurance, anddeductible for the prescription drug and/or perform a DUR (drugutilization review). Other communications between the order verificationsubsystem 302 and the benefit manager device 102 may be performed for avariety of purposes.

The order control subsystem 304 controls various movements of thecontainers and/or pallets along with various filling functions duringtheir progression through the system 100. In some implementations, theorder control subsystem 304 may identify the prescribed drug in one ormore than one prescription orders as capable of being fulfilled by theautomated dispensing device 214. The order control subsystem 304 maydetermine which prescriptions are to be launched and may determine thata pallet of automated-fill containers is to be launched.

The order control subsystem 304 may determine that an automated-fillprescription of a specific pharmaceutical is to be launched and mayexamine a queue of orders awaiting fulfillment for other prescriptionorders, which will be filled with the same pharmaceutical. The ordercontrol subsystem 304 may then launch orders with similar automated-fillpharmaceutical needs together in a pallet to the automated dispensingdevice 214. As the devices 206-232 may be interconnected by a system ofconveyors or other container movement systems, the order controlsubsystem 304 may control various conveyors: for example, to deliver thepallet from the loading device 208 to the manual fulfillment device 216from the literature device 228, paperwork as needed to fill theprescription.

The order tracking subsystem 306 may track a prescription order duringits progress toward fulfillment. The order tracking subsystem 306 maytrack, record, and/or update order history, order status, etc. The ordertracking subsystem 306 may store data locally (for example, in a memory)or as a portion of the order data 118 stored in the storage device 110.

Intervention System Block Diagrams

Returning to FIG. 1 , the member data 120 may also include health dataabout the user, such as conditions the user is diagnosed with, such ashypertension, high cholesterol, or diabetes. An intervention device 400obtains data from the storage devices 110 and may communicate with theuser devices 108, specialist devices 404, and client devices 408 vianetworks 104. The specialist devices 404 may include pharmacists andpharmacist technicians that may execute interventions, such as placingphone calls to users. The client devices 408 may be operated by clients,such as representatives of health insurers.

In FIG. 4 , an example implementation of the intervention device 400includes a care gap modeling circuit 412 and an intervention modelingcircuit 416. The care gap modeling circuit 412 develops a modelindicating a likelihood of risk for a user, such as a risk of a gap incare. These estimations are provided to an intervention managementcircuit 420. The intervention modeling circuit 416 develops a model ofthe likelihood of user engagement with interventions in general orspecific interventions. These intervention engagement data are providedto the intervention management circuit 420.

The intervention modeling circuit 416 may receive, as inputs to themodel, the claims data 122 for a specific user, demographic data of theuser, contact data of the user, and health data of the user. The caregap modeling circuit 412 may also receive these inputs—for illustrationpurposes, the care gap modeling circuit 412 is shown in FIG. 4 asreceiving just the claims data 122. Contact data may include whetherparticular forms of contact (such as an email address) are present andmay also include more granular information, such as the domain of theemail address. Demographic data may include, as examples, age andgender.

The claims data 122 may indicate historical data of when prescriptionswere filled and may include information about intervals between refills.The intervention modeling circuit 416 also receives information aboutpast interventions from an intervention data store 424. The interventiondata store 424 tracks prior interventions, including email campaigns,phone calls, etc.

The intervention data store 424 may also store information aboutintervention outcomes. These outcomes may include verbal feedbackprovided during phone call and may also include objective successmeasures from an intervention assessment circuit 428. The interventionassessment circuit 428 may receive the order data 118 and determinewhether successes resulted from interventions.

For example, if an order is placed within 30 days of intervention, thatmay be classified as a success by the intervention assessment circuit428. In various implementations, the business logic used by theintervention assessment circuit 428 to assess the success or failure ofan intervention may be established by the business logic of theorganization making the intervention. For example, an intervention thatprovides a discount for mail-order prescriptions may assess success onlyin response to receiving a mail-order script and not a retail script.

The intervention management circuit 420 stores care gap expectation dataand intervention expectation data for users in a model output data store432. A client user interface 436, which may be implemented in a webportal, allows the client devices 408 to access some or all of theinformation related to the care gap expectation data and interventionexpectation data. The client user interface 436 may also provide accessto information from the intervention data store 424, such as a list ofpast interventions for a user of interest.

The intervention management circuit 420 determines whether interventionis needed for a user and also determines the appropriate intervention.The intervention management circuit 420 assigns interventions to, in theexample of FIG. 4 , an automated intervention circuit 440 or aspecialist management circuit 444. The automated intervention circuit440 transmits automated interventions to users, such as emails,prerecorded calls, etc. The automated intervention circuit 440 may relyon contact data from the member data 120 and logs the interventions inthe intervention data store 424.

The specialist management circuit 444 schedules specialist interactions.For example, specialists may include pharmacists, pharmacy technicians,and call center operators. The specialist management circuit 444 mayprioritize and schedule specialist interventions. A specialist userinterface 448, which may be implemented as a web portal, indicates tospecialists using specialist devices 404 which users to contact. Forexample, a list of phone numbers to call may be presented to aspecialist operating one of the specialist devices 404. The specialistuser interface 448 allows the specialist to notate whether and whatreaction was provided by the user and may also provide a script for thespecialist to follow.

For example purposes only, FIGS. 9A-9E show example scripts that may bepresented by the specialist user interface 448. The scripts arepresented in flowchart form and may present each rectangle to thespecialist as the specialist makes choices such as yes or no. Additionalinformation can be found in U.S. Pat. No. 9,147,163, titled “Methods andsystems for improving therapy adherence” and issued Sep. 29, 2015, theentire disclosure of which is incorporated by reference.

In FIG. 5 , an example implementation of the intervention modelingcircuit 416 includes a general intervention circuit 504 that is agnosticto channel of intervention (such as phone call versus text message). Inthis example, the intervention modeling circuit 416 also includeschannel-specific intervention circuits 508-1 . . . 508-n (collectively,channel-specific intervention circuits 508). Each of thechannel-specific intervention circuits 508 may be specific to a singlechannel of intervention or a class of intervention. For example, oneclass of intervention may be real-time, which includes a phone call oronline chat, while a second class of intervention includes aunidirectional intervention such as an email, letter, or text message.

An ensemble circuit 512 selects from among the general interventioncircuit 504 and the channel-specific intervention circuits 508 andoutputs one or more intervention likelihoods. For example, the ensemblecircuit 512 may output likelihoods of the user engaging with anintervention for each type of intervention. In other implementations,the ensemble circuit 512 may output a single likelihood of engagement bythe user with any executed intervention. The ensemble circuit 512 mayselect which of the channel-specific intervention circuits 508, if any,are applicable to the user and ignore outputs of those that are notapplicable. The ensemble circuit 512 may calculate a weighted average ormay select a most applicable output when generating a single likelihoodvalue.

In other implementations, an example implementation of the interventionmodeling circuit 416 includes a general intervention circuit 604 that isagnostic to user condition. Meanwhile, the intervention modeling circuit416 also includes condition-specific intervention circuits 608-1 . . .608-n (collectively, condition-specific intervention circuits 608) thatare specific to conditions or class of conditions that a user may havebeen diagnosed with.

For example, a first class of condition may include chronic conditionssuch as diabetes or hypertension. A second class of condition mayinclude acute conditions such as viral or bacterial infections. Anensemble circuit 612 selects and/or combines outputs of the generalintervention circuit 604 and the condition-specific interventioncircuits 608. The ensemble circuit 612 may generate and output a singleintervention likelihood indicating how likely it is that the user willengage with an executed intervention. For example, the interventionlikelihood may be a value from 0 to 1, with a 1 indicating that the useris 100% likely to take action in response to an intervention beingexecuted (for example, a text message being sent or a call beingplaced).

Each of the models in the circuits 504, 508, 604, and 608 may beimplemented using a logistic regression classifier based on determinedfeature vectors. In various implementations, some or all of the modelsin the circuits 504, 508, 604, and 608 may be implemented using a deeplearning architecture (such as convolutional neural networks) thatreceive raw data not transformed into feature vectors. In variousimplementations, the deep learning architecture may essentially performintrinsic feature engineering, ignoring data that is not correlated withengagement outcomes.

In FIG. 7 , example operation of model creation and maintenance isshown. Control begins at 704, where for a given population (such as theusers corresponding to a particular health plan), data regardinginterventions that have been executed in the past is obtained. At 708,control assesses success rates of interventions from the priorintervention data. For example, as discussed above, success may bedefined by the manager of the prior interventions. At 712, controlobtains contact data, demographic data, and health data for the selectedpopulation. At 716, control segregates the population into mutuallyexclusive sets: a training set of users and a verification set of users.

At 720, control determines principal components for the features of thetraining set. In various implementations, there may be a large number offeatures (sometimes called variables) available for the population,which may increase the processing and storage demands of the machinelearning models. In addition, creating feature vectors with a largenumber of features, some of which may not be pertinent, may actuallydecrease the accuracy of the model.

The number of features (also referred to as dimensionality) may bereduced using a variety of techniques. As one example only, hundreds offeatures may be narrowed down to thirty or forty features. The techniquedescribed in FIG. 7 is principal component analysis (PCA). Specifically,PCA is used to rank order the available features so that a subset can beselected for inclusion in feature vectors. The subset is not necessarilya proper subset—in various implementations, some situations may lead tothe selected features including all of the available features.

Traditionally, PCA is used to identify principal components, which arecombinations of features (often weighted averages of the constituentfeatures), so that feature vectors can be created based on thoseprincipal components. That traditional approach is applicable to thepresent disclosure. However, in the approach described here with respectto FIG. 7 , PCA is used to quantify how well features are correlatedwith a target space. These quantities allow the features to be rankedand the highest-ranked features are themselves then selected forinclusion in feature vectors.

The principal components are formed from the features using atransformation matrix. In other words, each principal component is aweighted average of all of the features—each element of thetransformation matrix describes the weight of a certain feature ingenerating a certain principal component. In various implementations,the transformation matrix is a square matrix, meaning that the number ofprincipal components is equal to the number of features. However, someof the principal components may have very little significance(contribution to variability) and so may be ignored to reducedimensionality.

In general terms, principal components explain variance. Features thatare highly correlated will be highly weighted in a single principalcomponent. In PCA, the highest-variance-explaining component is thefirst principal component, the second-highest-variance-explainingcomponent is the second principal component, etc.

As a simplistic example, consider a setting in which three features arepresent (named x, y, and z) and therefore PCA defines three principalcomponents (pc1, pc2, and pc3). For this example, assume that thetransformation matrix derived by PCA is as follows:

TABLE 1 pc1 pc2 pc3 x 0.9 0.3 0.2 y 0.05 0.7 0.05 z 0.05 0 0.75

The transformation matrix describes how a point in feature space can betransformed to a point in principal component space, where dimension pc1for a point in principal component space is formed by adding 0.9 timesthe x dimension of the point in feature space, 0.05 times the ydimension of the point in feature space, and 0.9 times the z dimensionof the point in feature space. Note that the weights in each column sumto one.

The eigenvalue of each principal component is an indicator of how muchvariance the principal component explains. Principal components with aneigenvalue of less than a threshold are ignored for the followinganalysis. In the implementation of FIG. 7 , the threshold is 1.0. Forexample only, assume that the eigenvalues of pc1, pc2, and pc3 are 1.5,1.2, and 0.7, respectively. Using a threshold of 1.0, pc3 is excludedfrom further consideration. This leaves the following reduced matrix:

TABLE 2 pc1 pc2 x 0.9 0.3 y 0.05 0.7 z 0.05 0

Next, at 722, the principal component with which each feature is moststrongly correlated is identified. Feature x is most strongly correlatedwith pc1, feature y is most strongly correlated with pc2, and feature zis most strongly correlated with pc3. This may be represented in thetransformation matrix as follows, by only retaining values for items inthe matrix showing the strongest correlation for each feature.

TABLE 3 pc1 pc2 x 0.9 y 0.7 z 0.05

At 724, control performs a regression analysis for each of the sets offeatures associated with the remaining principal components. Theselection criteria for the regression analysis is to determine whichfeatures are more correlated with the model target (for example,likelihood of success of intervention). The first regression test(corresponding to pc1) would include features x and z as predictors,while the second regression test (corresponding to pc2) would includefeature y as a predictor.

The regression tests apply backward selection to only keep predictorsthat are most correlated with the target. Control calculates a Waldstatistic for each of x and z:

$\frac{\hat{\theta} - \theta_{0}}{{se}\left( \hat{\theta} \right)}$where {circumflex over (θ)} is the maximum likelihood estimate of thefeature, θ₀ is the target of the model (again, the observed likelihoodof success of intervention), and se({circumflex over (θ)}) is thestandard error of the maximum likelihood estimate (MILE) of {circumflexover (θ)}. If the Wald statistic has a high value, then the feature isconsidered to have a significant contribution to the model prediction.

Next, control determines a statistical value (such as a p-value or Zstatistic) of the Wald statistic for each feature. If the statisticalvalue is greater than a threshold (for example, if the p-value isgreater than 0.05), the feature is excluded. After removing features,the regression test is re-run until no features are removed.

Consider the first regression test, which corresponds to pc1 andincludes features x and z. Control calculates a p-value for each offeature x and feature z. Assuming that the p-value for feature z isgreater than 0.05, feature z is removed. The regression test is re-run,this time on only feature x. If the p-value of feature x in thisiteration is greater than 0.05, feature x would be eliminated from usein feature vectors. In this case, however, assume that the p-value offeature x in this iteration is less than 0.05. As a result, no featureswould be removed, and the first regression test would be finished.

Following all of the regression tests, the remaining sets of features(which may be the empty set) from each of the regression tests arecombined together. And then another regression test is performed on thecombination. Backward selection is used to eliminate features from thiscombination. The backward selection may use p-values and either the samethreshold (0.05) or a different threshold. Features with p-valuesfalling above the threshold are removed at each iteration until nofeatures are removed.

The resulting reduced set of features following backward selectiondetermines how feature vectors are created. At 726, control createsfeature vectors for the population (both the training set and theverification set) based on the selected features.

At 728, control trains a machine learning model, such as a logisticregression classifier, using selected feature vectors of the trainingset. At 730, control applies the model to feature vectors of users inthe verification set and determines the likelihood of engagement for theusers of the verification set. At 732, control compares the predictedengagement of the user's in the verification set with actual historicalsuccess rates for the users within the verification set. If the modeloutput is within a predetermined threshold of the actual historical data(such as the average absolute value of deviation being less than apredetermined limit), control transfers to 736; otherwise, controltransfers to 740.

At 740, control adjusts the model, such as by adjusting which featuresare included in the feature vectors or by adjusting other modelparameters. Control then returns to 720. Meanwhile, at 736, controlwaits for additional data to be received. Once additional data isreceived, control transfers to 744. At 744, control determines featurevectors for the additional data. At 748, control trains the model withthe determined feature vectors and returns to 736.

To determine feature vectors at 726, the present disclosure may firstreduce the number of features. For each of the variables, especially forcontinuous variables, there may be nonlinear aspects with respect to thetarget. So, a variable can be converted into a set of bins. For example,a quantile cut means defining bins such that approximately 5% of thedata falls into each bin. In other implementations, the quantile cutmeans that up to 5% of the data falls into each bin.

As one example, consider a variable such as age, which may range from18-85 and be approximately Gaussian with a mean around 30 or 40. Whenrank-ordering the data with respect to age, take the first 5% of thepopulation (the youngest 5%) and define a bin accordingly. Then take thenext 5% of the population and define a second bin.

After defining the 20 bins (or some other number if not using a quantilecut), combine all of the population members who do not have a value forthe variable into a missing value bin.

Then, determine a probabilistic score (which takes values between 0and 1) for the target for each bin. For example, determine a percentageindicating how likely the population within each bin is to respond toengagement and/or determine a percentage indicating the adherence of thepopulation within that bin. Then a logit can be calculated from theprobabilistic score (p) as follows:

${{logit}(p)} = {{\log\left( \frac{p}{1 - p} \right)}.}$With a missing value bin and using a quantile cut, there are now 21bins, each with a logit.

Each feature can therefore be converted into a logit that is fed into alogistic regression model instead of raw values. As one example, an agefeature is presented not as the number of years but as the calculatedlogit. A stepwise selection method can then reduce the set of featuresdown to those that are most correlated with the target—as one example,reducing the number of features from 60 to 8. Then the reduced set offeatures are used to train the model in 728.

The stepwise selection method may be implemented with a combination offorward selection and backward selection. In backward selection,variables are present in the model and at each step variables areanalyzed to see whether they are statistically significant—if not, theyare removed as features. Meanwhile, forward selection starts with nofeatures and begins adding only the most statistically significantfeatures.

At each step, control may test whether a variable can be added andwhether a variable can be removed. For each variable, control calculatesa chi-squared value (or, analogously, a p value) from either the Rao orWald distribution. For example, in a particular step of the stepwiseselection method, control calculates a Rao statistic for each variablenot yet in the model (forward selection), expressed as a p value. If thep value of a variable is less than a predetermined threshold for forwardselection (alpha threshold), the variable is a candidate for addition tothe model. If no variables have p values lower than the alpha threshold,then no variable is added to the model at that step. In variousimplementations, if there are multiple variables with p values lowerthan the alpha threshold during a particular step of the stepwiseselection method, only the variable with the largest chi-squared value(lowest p value) is added.

Meanwhile, control calculates a Wald statistic for each variable alreadyin the model (backward selection), expressed as a p value. If the pvalue of a variable currently in the model is greater than apredetermined threshold for backward selection (beta threshold), thenthe variable is removed from the model. In various implementations, thevalues of the alpha and beta thresholds may be the same (one numericalexample is 0.05). If no variables have p values higher than the betathreshold, then no variable is removed from the model at that step. Invarious implementations, if there are multiple variables with p valuesgreater than the beta threshold during a particular step of the stepwiseselection method, only the variable with the smallest chi-squared value(highest p value) is removed.

In light of the added or removed variables, control refits the logisticregression and performs another step of forward and backward selection.This process is continued until a particular criterion or criteria arereached. For example, the process may be repeated until a specificnumber of features (as a numerical example, 8) are in the model, untilthere are no features left to add to the model, after a predeterminednumber of iterations, etc. The goal is to develop a parsimonious model,which has as few features as possible while still performing wellfollowing training (as measured with test data).

In FIG. 8 , intervention analysis for a selected user begins at 804. Forexample, intervention analysis may be determined on a periodic basis,such as once per day, once per weekday, once per week, etc. In variousimplementations, the period may be selected by the client, such as thehealth insurer of the selected user.

At 804, for the selected user, control obtains prior intervention data.At 808, control obtains contact data, demographic data, and health data.At 812, control prepares a feature vector for the selected user based onthe obtained data. At 816, control inputs the data to an interventionmodel, such as the intervention modeling circuit 416, and stores theintervention expectation data for the selected user (for example, intothe model output data store 432 of FIG. 4 ). At 820, control inputs datainto a care gap model and stores care gap expectation data for the user(for example, into the model output data store 432 of FIG. 4 ).

At 824, control determines whether the care gap expectation data exceedsa threshold. If so, control transfers to 828; otherwise, intervention isnot required and control ends. At 828, control determines whetherchannel-specific intervention expectation data is available. If so,control transfers to 832; otherwise, control transfers to 836.

At 832, control weights the channel-specific intervention expectationdata by the respective costs of the intervention. As an example, thecost of a telephone call from a specialist is substantially higher thanthe cost of a text message. Control continues at 840, where controlselects an intervention for the user based on the weighted interventionexpectation data and the care gap expectation data. For example, theabove-described two-by-two grid or a combined score, also describedabove, may be employed. Control then continues at 844.

At 844, the selected intervention is added to the appropriate queue. Forexample, a queue for an automated intervention may be maintained by theautomated intervention circuit 440 of FIG. 4 . The automated queue mayinclude a time and date at which the intervention should be executed(transmitted). For an intervention involving a specialist, thespecialist management circuit 444 of FIG. 4 may maintain a queue forinterventions to be executed by specialists. Following 844, controlends.

At 836, channel-specific intervention expectation data is not availableand therefore an intervention is selected based on the singleintervention expectation data and the care gap expectation data. Inaddition, the intervention may be selected according to the cost of theintervention. The intervention selection may be performed using, forexample, the above-described two-by-two table. Control then continues at844.

Channel-Specific Engagement Machine Learning Architecture

In FIG. 10 , an example implementation of a channel-specificintervention device 900 includes a care gap modeling circuit 412 and anintervention modeling circuit 916. The care gap modeling circuit 412develops a model indicating a likelihood of risk for a user, such as arisk of a gap in care. These estimations are provided to an interventionmanagement circuit 920. The intervention modeling circuit 916 develops amodel of the likelihood of user engagement with channel-specificinterventions. The intervention engagement data is provided to theintervention management circuit 920.

The intervention modeling circuit 916 may receive, as inputs to themodel, the claims data 122 for a specific user, demographic data of theuser, contact data of the user, and health data of the user. The caregap modeling circuit 412 may also receive these inputs—for illustrationpurposes, the care gap modeling circuit 412 is shown in FIG. 10 asreceiving just the claims data 122. Contact data may include indicationsof whether particular forms of contact (such as an email address) arepresent, and may also include more granular information, such as thedomain of the email address. Demographic data may include, as examples,age and gender.

The claims data 122 may indicate historical data of when prescriptionswere filled and may include information about intervals between refills.The intervention modeling circuit 916 also receives information aboutpast interventions from an intervention data store 424. The interventiondata store 424 tracks prior interventions, including email campaigns,phone calls, etc.

The intervention data store 424 may also store information aboutintervention outcomes. These outcomes may include verbal feedbackprovided during a phone call and may also include objective successmeasures from an intervention assessment circuit 428. The interventionassessment circuit 428 may receive the order data 118 and determinewhether successes resulted from interventions.

For example, if an order is placed within 30 days of intervention, thatmay be classified as a success by the intervention assessment circuit428. In various implementations, the business logic used by theintervention assessment circuit 428 to assess the success or failure ofan intervention may be established by the business logic of theorganization making the intervention. For example, an intervention thatprovides a discount for mail-order prescriptions may assess success onlyin response to receiving a mail-order script and not a retail script.

The intervention management circuit 920 stores care gap expectation dataand intervention expectation data for users in a model output data store432. A client user interface 436, which may be implemented in a webportal, allows the client devices 408 to access some or all of theinformation related to the care gap expectation data and interventionexpectation data. The client user interface 436 may also provide accessto information from the intervention data store 424, such as a list ofpast interventions for a user of interest.

The intervention management circuit 920 determines whether interventionis needed for a user and also determines the appropriate intervention.The intervention management circuit 920 assigns interventions to, in theexample of FIG. 10 , an automated intervention circuit 440 or aspecialist management circuit 444. The automated intervention circuit440 transmits automated interventions to users, such as emails,prerecorded calls, etc. The automated intervention circuit 440 may relyon contact data from the member data 120 and logs the interventions inthe intervention data store 424.

The specialist management circuit 444 schedules specialist interactions.For example, specialists may include pharmacists, pharmacy technicians,and call center operators. The specialist management circuit 444 mayprioritize and schedule specialist interventions. A specialist userinterface 448, which may be implemented as a web portal, indicates tospecialists using specialist devices 404 which users to contact. Forexample, a list of phone numbers to call may be presented to aspecialist operating one of the specialist devices 404. The specialistuser interface 448 allows the specialist to notate whether and whatreaction was provided by the user and may also provide a script for thespecialist to follow.

In FIG. 11 , an example implementation of the intervention modelingcircuit 916 includes channel-specific intervention circuits 1008-1 . . .1008-n (collectively, channel-specific intervention circuits 1008). Eachof the channel-specific intervention circuits 1008 may be specific to asingle channel of intervention or a class of intervention. For example,one class of intervention may be real-time, which includes a phone call(e.g., with a pharmacist or physician) or online chat, while a secondclass of intervention includes a unidirectional intervention such as anautomated call, an email, a letter, or a text message.

Each channel-specific intervention circuit 1008 outputs an interventionlikelihood for the channel intervention corresponding to that circuit1008 (e.g., a likelihood that a patient will engage, become adherent,etc., if the patient is contacted via the channel interventioncorresponding to that circuit 1008). The ensemble circuit 512 may outputlikelihoods of the user engaging with an intervention for each type ofintervention.

For example, the intervention likelihood may be a value from 0 to 1,with a 1 indicating that the user is 100% likely to take action inresponse to a channel-specific intervention being executed (for example,a text message being sent or a call being placed). In variousimplementations, the intervention likelihood may be scored in one ormore categories or ranges (e.g., high, medium or low), etc.

Each model of the channel-specific intervention circuits 1008 may beimplemented using a logistic regression classifier based on determinedfeature vectors. In various implementations, some or all of the modelsmay be implemented using a deep learning architecture (such asconvolutional neural networks) that receives raw data that is nottransformed into feature vectors. In various implementations, the deeplearning architecture may essentially perform intrinsic featureengineering, ignoring data that is not correlated with engagementoutcomes.

In FIG. 12 , an example implementation of the intervention managementcircuit 920 includes an intervention selection circuit 1040. Theintervention selection circuit receives member data 120 for a patient,including demographic data and health data. The intervention selectioncircuit 1040 also receives intervention likelihoods from theintervention modeling circuit 916.

For example, the channel-specific intervention likelihoods (e.g.,channel-specific intervention expectations) from each of thechannel-specific intervention circuits 1008 may be supplied from theintervention modeling circuit 916 to the intervention selection circuit1040, so the intervention selection circuit 1040 has likelihoods ofengagement for the patient via each of the channels (e.g., a likelihoodof engaging the patient via an automated call, a likelihood of engagingthe patient via an email, a likelihood of engaging the patient via aletter, a likelihood of engaging the patient via a live call from aspecialist such as a physician or pharmacist, etc.). In variousimplementations, a general or channel-agnostic likelihood of engagementfor the patient overall may be supplied to the intervention selectioncircuit 1040.

As shown in FIG. 12 , objective targets 1044 are supplied to theintervention selection circuit 1040. The objective targets 1044 mayrepresent goals that an administrator of the channel-specific engagementmachine learning architecture is trying to achieve. For example, thesystem administrator (e.g., a health plan provider) may offer aguarantee that an average of patients within a group (e.g., employees ofa company) that use the channel-specific engagement machine learningarchitecture will be at least eighty percent adherent, a guarantee thatat least fifty percent of identified non-adherent patients will becomeadherent, a guarantee that lab data (e.g., A1C test results) for a groupof patients will be improved by a specified factor or will meet aspecified threshold, etc.

Objective tracking data 1048 is supplied to the intervention selectioncircuit 1040 to monitor progress towards the targets, such asidentifying how many patients are adherent relative to a guarantee,monitoring relevant lab test data for patients, etc. When the objectivetracking data 1048 indicates that a group of patients are below thetarget, this may suggest that higher intervention likelihood channelsshould be used to increase the chance of patient engagement to returncloser to the target. Objective tracking data 1048 that suggests a groupof patients overall are above a target, may suggest that lowerlikelihood intervention channels may be used to free up interventionchannels for other patients.

The intervention selection circuit 1040 also receives channel cost data1052. The channel cost data 1052 allows the intervention selectioncircuit to weight the intervention likelihoods from the models of thechannel-specific intervention circuits 1008 of the intervention modelingcircuit based on how expensive each channel intervention is. Forexample, while live calls from a specialist may frequently have a higherintervention likelihood than an email, live calls from a specialist arealso typically much more expensive than an email. Each live call from aphysician may cost ten dollars, twenty dollars, etc., of the physician'stime, while an automated call may use resources that cost pennies orless individually, and emails or text messages even less than that.

Similar to the channel cost data 1052, channel capacity data 1056 mayalso be supplied to the intervention selection circuit 1040. As anexample, a pharmacist may only be able to complete one hundred livecalls in a single day, while an automated system may be able to placethousands, and email or text messages may have even more capacity thanthat.

The intervention selection circuit 1040 may weight the interventionlikelihoods from each different model of the channel-specificintervention circuits 1008 according to the relative costs and availablecapacity, in order to determine prioritization of which channels to usefor which patients. For example, live calls may be prioritized forpatients having very low intervention likelihoods in the other channels,while patients that have more equal likelihoods for each channel may beassigned to email or text message intervention channels to shift costsor allocate capacity to patients where the more expensive channels(e.g., automated calls and particularly live calls) may have greaterrelative effect in generating patient engagement. Although the presentexample uses four different intervention channels, other embodiments mayinclude more or less channel-specific intervention circuits 1008 andassociated models, other types of intervention channels, etc.

The intervention selection circuit 1040 may incorporate features fromthe member data 120 in order to prioritize selection of interventionchannels for engaging a patient. The member data 120, interventionselection circuit 1040, etc., may identify or determine a cost to thepatient to refill a prescription. For example, the circuit 1040 mayincorporate a cost of the prescription, a co-pay or other informationfrom a health plan of the patient, etc. Actions that will require ahigher cost to the patient may suggest that a higher likelihoodintervention channel should be used in an attempt to prevent the patientfrom delaying a refill further due to costs.

Similarly, the member data 120 may include pill burden information, suchas a number of pills that the patient is currently taking. If the pillburden indicates a high number of current pills, this may suggest that ahigher likelihood intervention channel should be used because thepatient may have forgotten one of the many pills, the patient may behesitant to refill the prescription and add to the number of pills thatthey must take, etc.

In various implementations, the member data 120 may include comorbidityinformation. If the patient has multiple diseases, this may suggest thata higher likelihood intervention channel should be used because thepatient is at a higher risk of an adverse event such as ahospitalization event, a non-adherence event, a serious health event,etc. The patient comorbidity condition, pill burden, refill cost, etc.,may be considered as user adherence factors.

The intervention selection circuit 1040 output an intervention selectionthat prioritizes one or more of the intervention channels based on themember data 120, the objective targets 1044, the objective tracking data1048, the channel cost data 1052, and the intervention likelihoods fromthe intervention modeling circuit 916. The intervention selectioncircuit 1040 may include one or more models that balance the patient'sintervention likelihoods from each of the channel-specific interventioncircuits 1008, constraints such as costs and channel capacity, objectivetargets 1044, patient comorbidity data, etc., in order to optimize theselection of an intervention channel for the patient. The interventionselection circuit 1040 may include any suitable model(s) forprioritizing the intervention channel selection, such as a linearprogramming model, a logistic regression model, a random forest model, adecision tree model, a heuristic model, etc.

Channel-Specific Model Intervention Selection

In FIG. 13A, intervention analysis for a selected user begins at 1302.For example, intervention analysis may start in response to a gap incare trigger for an identified user, such as one or more business rulesthat identify a user having an outstanding prescription refill that isbeyond an expected fill date by a threshold time period, etc.Alternatively, or in addition, intervention analysis may be determinedon a periodic basis, such as once per day, once per weekday, once perweek, etc. In various implementations, the period may be selected by theclient, such as the health insurer of the selected user.

At 1302, control obtains prior intervention data, contact data,demographic data, and health data for the identified user (such as fromthe member data 120). At 1304, control determines whether a time periodsince the last intervention is greater than a threshold. If not, controlproceeds to 1306 to wait until the threshold time since the lastintervention is reached. For example, a minimum threshold time may beset between interventions, so a user is not receiving interventionengagements every day, etc.

If the time since the last intervention is greater than the threshold at1304 (or if this is the first intervention), control proceeds to 1308 todetermine a channel-agnostic likelihood of engagement. As describedabove, the channel-agnostic likelihood of engagement may indicate ageneral likelihood of a patient responding to an intervention using anychannel.

If the channel-agnostic likelihood is less than a threshold at 1310,control proceeds to 1312 to use a low engagement intervention processand attempt to identify a reason for the user's lower engagement. Forexample, instead of using the basic intervention channels of thearchitecture, a specialist may identify other types of interventionsthat may be tried, additional patient data may be analyzed to identifyalternative engagement outreach, etc.

In various implementations, control may attempt to identify a reason forthe user's low overall engagement by storing and analyzing the user'smember data 120. Identified or predicted reasons may be saved to improvethe model, to improve intervention channel options, etc.

At 1314, control determines a channel-specific likelihood of engagementfor n channels, which may involve use of the channel-specificintervention circuits 1008 of the intervention modeling circuit 916. Asan example, the n channels may include a live call, an automated call,an email, a text message, etc.

Control analyzes user data to determine a risk score at 1316. The riskscore may be indicative of a likelihood of a future adverse event forthe user, such as a hospitalization event, a non-adherence event, aserious health event, etc. The user data may include data on acomorbidity condition for the user, a length or frequency ofnon-adherence, a type of disease of the user, past claims or medicalhistory data, etc. If the risk score is greater than a threshold at1318, control proceeds to 1320 to schedule an intervention using thechannel with the highest likelihood of engagement. For example, if alive call has the highest likelihood of engagement for a user, and theuser has a high risk of a future adverse event, a live call may bescheduled prior to weighing costs, capacity, etc., because of the user'shigh risk and the desire to engage with the user as much as possible toavoid the adverse event.

If the risk score is below the threshold, control proceeds to 1322 inFIG. 13B. At 1322, control determines a cost for each interventionchannel, and adjusts the cost for each intervention channel based oncapacity at 1324. Live calls may be assigned a cost based on the salary,hourly wage, etc., of a specialist that performs the live calls, dividedby an average number of calls that the specialist can perform during aspecified time period. The capacity is also taken into account where ashortage of live call specialists may adjust the cost upwards for livecall interventions, while a large supply of available live callspecialists may adjust the cost downwards for live call interventions.Similar cost and capacity analysis may be performed on otherintervention channels, and the costs and capacities may be analyzed andadjusted using any suitable approaches.

At 1326, control identifies targets that are relevant to a user. Theuser may be part of a company health plan that has been offered aguarantee of at least eighty percent adherence for their members, theuser may be part of a group of diabetes patients where a guarantee hasbeen made to keep the average A1C test results for the group below aspecified threshold (e.g., below 8), etc.

Control then proceeds to 1328 to determine measures of progress towardrelevant targets. This may include analyzing objective targets 1044 andobjective tracking data 1048. As an example, if a target adherence levelfor a group that the user belongs to is eighty percent, a measure ofonly seventy percent adherence for the group would indicate that thegroup is behind the target and therefore higher likelihood interventionchannel(s) should be scheduled for the user to increase the progresstoward the target. In contrast, a measure of ninety percent adherencefor the group would indicate that the group is ahead of the target andtherefore lower likelihood intervention channel(s) may be scheduled toallow for allocation of higher cost channels to other users.

At 1330, control determines whether any of the measures of progress fortargets relevant to the user are less than a minimum threshold. If so,control proceeds to 1332 to schedule an intervention using the channelthat has the highest likelihood of engagement. In the above example, ifeighty percent adherence is the target value and seventy percent is aminimum threshold, a current group measure of only sixty percentadherence may require reaching out to the user with the highestlikelihood channel in an attempt to more quickly raise the averageadherence for the group back to the target or at least above the minimumthreshold.

If there are not any measures of progress towards target(s) that arebelow a minimum threshold, control proceeds to 1334 to determine anengagement importance metric based on measures of progress. For example,if the measure of progress is below the target, the engagementimportance metric may have a higher value because it is more importantto engage the user to bring the measure of progress back up for therelevant group. Conversely, if the measure of progress is above thetarget, the engagement importance metric may have a lower value becauseit is less important to engage the user as the measure of progress isalready high.

At 1336, control discounts a cost of each channel based on theengagement importance metric. If the engagement importance metric has ahigh value (e.g., it is more important than average to engage the userto try to raise the measure of progress towards the target), the cost ofeach channel may be discounted by a larger amount because the system ismore willing to prioritize the expensive channels due to the need toraise the measure of progress. In contrast, if the engagement importancemetric has a low value (e.g., it is less important than average toengage the user because the measure of progress is already above thetarget), the cost of each channel may not be discounted at all, or byonly a small amount, because the system has a lower priority forengaging this user and may prefer to allocate higher cost resources tousers in groups having lower measures of progress towards targets.

Control then proceeds to 1338 to weight the channel-specific likelihoodof engagement for each channel, based on the discounted cost and theuser data. The weighting may take into account any user data, includingmember data 120, such as a refill cost to the user, a pill burden of theuser, comorbidity information about the user, etc. This can be combinedwith the discounted costs determined at 1336 (which take into accountresource and cost constraints for the different channels, in addition tomeasures of progress toward relevant cost), in order to adjust weightsof the channel-specific likelihoods determined at 1314 (e.g.,intervention likelihoods from the channel-specific intervention circuits1008).

At 1340, control selects the channel with the highest weightedlikelihood of engagement. For example, the intervention selectioncircuit 1040 may include one or more models that perform the weightingadjustment of step 1338 (in addition to any other suitable elements offlow of FIGS. 13A and 13B), and the circuit 1040 may then select thechannel that has the highest weighted likelihood at 1340. The circuit1040 may include any suitable model(s), such as a linear programmingoptimizer model, a logistic regression model, a decision tree model, arandom forest model, a heuristic model, etc.

If the selected channel does not include multiple interventions at 1342,control proceeds to schedule the selected intervention at 1344. However,some channels may include multiple interventions within the singlechannel. For example, live or automated calls may follow differentpre-defined scripts depending on how the user responds during the call,emails or text messages may have different text options, etc.

If the selected channel has multiple intervention options at 1342,control proceeds to 1346 to determine a cost for each interventionoption within each channel. For example, a longer or more complex callscript may increase the likelihood of engaging a user even though thelonger or more complex call script is more costly due to the increasedtime for a specialist to be on the call. FIGS. 9A-9E illustrate examplecall scripts of various complexity and length. In some channels, thecost for each intervention option may be approximately the same (e.g.,an email or text message cost is similar regardless of the text of themessage).

At 1348, control adjusts the cost for each intervention option based onthe available capacity. At 1350, control discounts the cost of eachintervention option based on the engagement importance metric (e.g., theengagement importance metric determined at 1334). Adjusting anddiscounting the cost of each intervention option based on capacity andthe engagement importance metric may be similar to steps 1324 and 1336,but applied to each intervention option within a single channel.

At 1352, control weights, for each intervention option within theselected channel, the likelihood of engagement based on the determineddiscounted cost and the user data. Control then selects the interventionoption with the highest weighted likelihood of engagement at 1354. Forexample, multiple variations of scripts, message texts, etc., may begenerated and test for success rates over time (e.g., success ofproducing user engagement, etc.).

In addition to determining a channel-specific likelihood of engagement,the channel-specific intervention circuits 1008 may also outputengagement likelihoods for each intervention option within a channel.Control (e.g., the intervention selection circuit 1040) then selects theintervention option with the highest likelihood of engagement at 1354and schedules the selected intervention option at 1344.

CONCLUSION

The foregoing description is merely illustrative in nature and is in noway intended to limit the disclosure, its application, or uses. Thebroad teachings of the disclosure can be implemented in a variety offorms. Therefore, while this disclosure includes particular examples,the true scope of the disclosure should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. It should be understood thatone or more steps within a method may be executed in different order (orconcurrently) without altering the principles of the present disclosure.Further, although each of the embodiments is described above as havingcertain features, any one or more of those features described withrespect to any embodiment of the disclosure can be implemented in and/orcombined with features of any of the other embodiments, even if thatcombination is not explicitly described. In other words, the describedembodiments are not mutually exclusive, and permutations of one or moreembodiments with one another remain within the scope of this disclosure.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Thephrase at least one of A, B, and C should be construed to mean a logical(A OR B OR C), using a non-exclusive logical OR, and should not beconstrued to mean “at least one of A, at least one of B, and at leastone of C.”

In the figures, the direction of an arrow, as indicated by thearrowhead, generally demonstrates the flow of information (such as dataor instructions) that is of interest to the illustration. For example,when element A and element B exchange a variety of information butinformation transmitted from element A to element B is relevant to theillustration, the arrow may point from element A to element B. Thisunidirectional arrow does not imply that no other information istransmitted from element B to element A. Further, for information sentfrom element A to element B, element B may send requests for, or receiptacknowledgements of, the information to element A. The term subset doesnot necessarily require a proper subset. In other words, a first subsetof a first set may be coextensive with (equal to) the first set.

In this application, including the definitions below, the term “module”or the term “controller” may be replaced with the term “circuit.” Theterm “module” may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuit(s) may implement wired or wireless interfaces thatconnect to a local area network (LAN) or a wireless personal areanetwork (WPAN). Examples of a LAN are Institute of Electrical andElectronics Engineers (IEEE) Standard 802.11-2016 (also known as theWIFI wireless networking standard) and IEEE Standard 802.3-2015 (alsoknown as the ETHERNET wired networking standard). Examples of a WPAN areIEEE Standard 802.15.4 (including the ZIGBEE standard from the ZigBeeAlliance) and, from the Bluetooth Special Interest Group (SIG), theBLUETOOTH wireless networking standard (including Core Specificationversions 3.0, 4.0, 4.1, 4.2, 5.0, and 5.1 from the Bluetooth SIG).

The module may communicate with other modules using the interfacecircuit(s). Although the module may be depicted in the presentdisclosure as logically communicating directly with other modules, invarious implementations the module may actually communicate via acommunications system. The communications system includes physicaland/or virtual networking equipment such as hubs, switches, routers, andgateways. In some implementations, the communications system connects toor traverses a wide area network (WAN) such as the Internet. Forexample, the communications system may include multiple LANs connectedto each other over the Internet or point-to-point leased lines usingtechnologies including Multiprotocol Label Switching (MPLS) and virtualprivate networks (VPNs).

In various implementations, the functionality of the module may bedistributed among multiple modules that are connected via thecommunications system. For example, multiple modules may implement thesame functionality distributed by a load balancing system. In a furtherexample, the functionality of the module may be split between a server(also known as remote, or cloud) module and a client (or, user) module.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of a non-transitory computer-readable medium are nonvolatilememory devices (such as a flash memory device, an erasable programmableread-only memory device, or a mask read-only memory device), volatilememory devices (such as a static random access memory device or adynamic random access memory device), magnetic storage media (such as ananalog or digital magnetic tape or a hard disk drive), and opticalstorage media (such as a CD, a DVD, or a Blu-ray Disc).

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium. Thecomputer programs may also include or rely on stored data. The computerprograms may encompass a basic input/output system (BIOS) that interactswith hardware of the special purpose computer, device drivers thatinteract with particular devices of the special purpose computer, one ormore operating systems, user applications, background services,background applications, etc.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language), XML (extensible markuplanguage), or JSON (JavaScript Object Notation), (ii) assembly code,(iii) object code generated from source code by a compiler, (iv) sourcecode for execution by an interpreter, (v) source code for compilationand execution by a just-in-time compiler, etc. As examples only, sourcecode may be written using syntax from languages including C, C++, C#,Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl,Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5threvision), Ada, ASP (Active Server Pages), PHP (PHP: HypertextPreprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

The invention claimed is:
 1. A computer-implemented method comprising: generating an intervention model by: determining principal components for features of a training set, associating each feature of the training set with a principal component, selecting features of the training set most highly correlated with principal components, performing a regression analysis on the selected features to determine a subset of the selected features that are most highly correlated with a model target, training a machine learning model based on the subset of the selected features, and saving the trained machine learning model as the intervention model; obtaining data related to a user, wherein: the data includes engagement data indicating successfulness of prior interventions with the user and each prior intervention with the user is associated with one of multiple engagement channels; supplying the obtained data as input to the intervention model to determine multiple channel-specific intervention expectations, wherein each channel-specific intervention expectation: corresponds to one of the multiple engagement channels and indicates a likelihood that the user will take action in response to an intervention being executed using the corresponding engagement channel; determining a likelihood of a gap in care for the user; and in response to the care gap likelihood being outside of a threshold: identifying a highest determined value of the channel-specific intervention expectations, selecting an intervention corresponding to the highest determined value of the channel-specific intervention expectation, and scheduling the selected intervention for execution.
 2. The method of claim 1 wherein: at least one of the multiple engagement channels includes multiple intervention options within the engagement channel; selecting the intervention includes selecting one of the multiple interventions within the engagement channel that has a highest intervention expectation among the intervention options; and scheduling the intervention includes scheduling the selected one of the multiple intervention options within the engagement channel.
 3. The method of claim 1 wherein: the intervention model includes a channel-agnostic intervention model that determines a general intervention expectation indicating a likelihood that the user will take action in response to any intervention being executed using any of the engagement channels and the method further comprises, in response to the general intervention expectation being below a specified threshold, initiating a low engagement intervention process and identifying at least one reason for low engagement of the user.
 4. The method of claim 1 further comprising, in response to determining that a time elapsed since a most recent intervention for the user is less than a specified delay threshold, waiting until the specified delay threshold has elapsed prior to selecting the intervention.
 5. The method of claim 1 further comprising: identifying one or more targets relevant to the user; determining a measure of progress toward at least one of the identified targets; determining an engagement importance metric based on the determined measure of progress; and weighting the channel-specific intervention expectations according to the determined engagement importance metric prior to selecting the intervention.
 6. The method of claim 5 wherein scheduling the intervention includes, in response to determining that the measure of progress is less than a specified minimum threshold, scheduling an intervention corresponding to the channel-specific intervention expectation that has a highest determined value prior to weighting.
 7. The method of claim 1 further comprising: determining a cost of engagement for each of the multiple engagement channels; determining a channel capacity for each of the multiple engagement channels; and weighting the channel-specific intervention expectations according to the determined costs of engagement and channel capacities prior to selecting the intervention.
 8. The method of claim 1 further comprising: determining at least one user adherence factor associated with the user and weighting the channel-specific intervention expectations according to the determined user adherence factor prior to selecting the intervention, wherein the at least one user adherence factor includes at least one of a prescription refill cost, a pill burden, and a comorbidity condition.
 9. The method of claim 8 wherein scheduling the intervention includes, in response to the comorbidity condition indicating a future adverse risk event that is higher than a specified threshold, scheduling the intervention corresponding to the channel-specific intervention expectation that has a highest determined value prior to weighting.
 10. The method of claim 1 further comprising verifying the trained machine learning model with a verification set prior to saving the trained machine learning model.
 11. A system comprising: memory hardware configured to store instructions and processing hardware configured to execute the instructions stored by the memory hardware, wherein the instructions include: generating an intervention model by: determining principal components for features of a training set, associating each feature of the training set with a principal component, selecting features of the training set most highly correlated with principal components, performing a regression analysis on the selected features to determine a subset of the selected features that are most highly correlated with a model target, training a machine learning model with the subset of the selected features, saving the trained machine learning model as the intervention model; obtaining data related to a user, wherein: the data includes engagement data indicating successfulness of prior interventions with the user and each prior intervention with the user is associated with one of multiple engagement channels; supplying the obtained data as input to the intervention model to determine multiple channel-specific intervention expectations, wherein each channel-specific intervention expectation: corresponds to one of the multiple engagement channels and indicates a likelihood that the user will take action in response to an intervention being executed using the corresponding engagement channel; determining a likelihood of a gap in care for the user; and in response to the care gap likelihood being outside of a threshold: identifying a highest determined value of the channel-specific intervention expectations, selecting an intervention corresponding to the highest determined value of the channel-specific intervention expectation, and scheduling the selected intervention for execution.
 12. The system of claim 11 wherein: at least one of the multiple engagement channels includes multiple intervention options within the engagement channel; selecting the intervention includes selecting one of the multiple interventions within the engagement channel that has a highest intervention expectation among the intervention options; and scheduling the intervention includes scheduling the selected one of the multiple intervention options within the engagement channel.
 13. The system of claim 11 wherein: the intervention model includes a channel-agnostic intervention model that determines a general intervention expectation indicating a likelihood that the user will take action in response to any intervention being executed using any of the engagement channels and the instructions include, in response to the general intervention expectation being below a specified threshold, initiating a low engagement intervention process and identifying at least one reason for low engagement of the user.
 14. The system of claim 11 wherein the instructions include, in response to determining that a time elapsed since a most recent intervention for the user is less than a specified delay threshold, waiting until the specified delay threshold has elapsed prior to selecting the intervention.
 15. The system of claim 11 wherein the instructions include: identifying one or more targets relevant to the user; determining a measure of progress toward at least one of the identified targets; determining an engagement importance metric based on the determined measure of progress; and weighting the channel-specific intervention expectations according to the determined engagement importance metric prior to selecting the intervention.
 16. The system of claim 15 wherein scheduling the intervention includes, in response to determining that the measure of progress is less than a specified minimum threshold, scheduling an intervention corresponding to the channel-specific intervention expectation that has a highest determined value prior to weighting.
 17. The system of claim 11 wherein the instructions include: determining a cost of engagement for each of the multiple engagement channels; determining a channel capacity for each of the multiple engagement channels; and weighting the channel-specific intervention expectations according to the determined costs of engagement and channel capacities prior to selecting the intervention.
 18. The system of claim 11 wherein: the instructions include: determining at least one user adherence factor associated with the user and weighting the channel-specific intervention expectations according to the determined user adherence factor prior to selecting the intervention and the at least one user adherence factor includes at least one of a prescription refill cost, a pill burden, and a comorbidity condition.
 19. The system of claim 18 wherein scheduling the intervention includes, in response to the comorbidity condition indicating a future adverse risk event that is higher than a specified threshold, scheduling the intervention corresponding to the channel-specific intervention expectation that has a highest determined value prior to weighting.
 20. The system of claim 11 wherein the instructions include verifying the trained machine learning model with a verification set prior to saving the trained machine learning model. 